People are utilizing portable computing devices for an increasing number and variety of tasks. One such task involves mapping a location such as the inside of a home or office utilizing such a portable device. When mapping the inside of a dwelling, for example, a user might utilize a portable computing device (such as a smart phone or tablet computer) to take multiple images of the interior of the dwelling. An application on the device then attempts to stitch together the images such that if a user is viewing a three-dimensional representation of the interior of the dwelling, the representation will not appear as a set of discrete images but rather a single continuous image representing the interior of the dwelling. In order to stitch the images together, there must be some way to relate images of adjacent regions of the interior. For example, the location of a doorway in one image can be used to determine the relative overlap between two images if that door also appears in the other image. In many cases, however, there will be images showing substantially blank walls, regions with repetitive patterns, or other such features that do not lend themselves to registration, as there maybe no features that can be correlated with an acceptable level of confidence. Further, since the user capturing the images will generally be holding the portable computing device in the user's hand and walking around to take the images, there generally will be no precise way to determine the location and direction in which the camera was facing when the image was captured.